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Abstract Body 


Background / Context: 

The Research Alliance for New York City Schools is committed to conducting rigorous, non- 
partisan research studies on topics that matter to the city’s public schools. One of the core 
functions of the Research Alliance is to conduct research in collaboration with other researchers 
at NYU, at other universities, and at other organizations across the city and country. Building 
diverse research teams enables us to conduct the highest quality studies with multi-disciplinary 
experts that reflect the complexity of challenges in education. 

While independent of the New York City Department of Education (DOE), the Research 
Alliance also works with DOE to identify important questions for research and to provide valid 
and reliable evidence to help solve problems and build capacity in schools throughout NYC. 
Recently, we have begun collaborating with the Department on the evaluation of a district- wide 
led initiative designed to improve the college and career readiness for young men of color. This 
panel will highlight the opportunities and challenges of our partnership as well as the operational 
and political realities of working with the city’s schools. By using this particular evaluation as an 
anchor, our research team and the project’s director at the DOE will present our process for site 
selection and recruitment; data collection and analysis; as well as reporting and dissemination of 
results to inform the development of the initiative. Below is background information on the 
initiative itself: 

In August, 2011, Mayor Michael R. Bloomberg announced the launch of the Young Men’s 
Initiative, a combination of new programs and policy reforms designed to mitigate the higher 
rates of poverty, incarceration, and unemployment among young Black and Latino men (Office 
of the Mayor, 2011). Of the $127 million invested in this effort, the New York City Department 
of Education (NYCDOE) will oversee a three-year, $30 million investment as part of the 
Expanded Success Initiative (ESI; NYCDOE, 2012). This program addresses a key goal of the 
larger initiative - reducing the achievement gap. In particular, the goal of the ESI is to invest 
deeply in a targeted group of schools to identify, implement, and learn from the best practices to 
close the gap between Black and Latino young men and their counterparts. The ESI intends to 
learn from schools that have demonstrated progress in closing the achievement gap in high 
school graduation rates and have these schools operate as laboratories to develop strategies that 
further close the gap in graduation and better prepare these young men for college and careers. 

Purpose / Objective / Research Question / Focus of Study: 

The Research Alliance for New York City Schools at New York University (Research Alliance) 
is undertaking a mixed-method, longitudinal evaluation of the Expanded Success Initiative (ESI). 
The Research Alliance’s evaluation will assess the implementation and impacts of the ESI in the 
40 schools that are selected for the initiative. The evaluation will address the following 
questions: 

• Implementation and Process Study. What services and supports are planned and 
implemented under the ESI? What impacts do ESI services and supports have on the 
school’s academic rigor, youth development practices, and culture? What challenges do 
schools confront as they implement the ESI services and supports and how do they 
address those challenges? 
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• Impact Study. What impacts do ESI services and supports have on attendance and 
absenteeism, academic performance, progress toward graduation and college readiness of 
Black and Latino young men? What impacts do ESI services and supports have on 
student social and emotional competencies, persistence, response to challenge, “grit,” and 
measures of “college knowledge” of Black and Latino young men? 

Setting: 

Both the implementation and impact studies will focus on the 40 ESI schools. In addition, the 
impact study will identify a group of equivalent comparison schools. 

Population / Participants / Subjects: 

Participants in ESI schools include principals and members of the ESI planning team. 

Participants in the comparison schools will include principals and staff members involved with 
programs that are similar to those intended to be provided through ESI. Students from both the 
ESI schools and comparison schools will participate in surveys. 

Intervention / Program / Practice: 

In the fall of 2011, the NYCDOE invited approximately 70 high schools that have shown success 
in graduating Black and Latino males to participate in a design challenge. The design challenge 
application asked schools to propose targeted supports in three inter-related areas: (1) academic 
coherence, (2) youth development, and (3) college and career access. From these proposals, the 
NYCDOE will select 40 schools to receive ESI resources to implement their plans starting in the 
fall of 2012 through the spring of 2016. Schools may propose and implement strategies and 
programs from each of the following areas: 

• Academic coherence includes curricular and performance task alignment with the 
Common Core Learning Standards. In addition, the integration of the technical 
knowledge and academic skills (e.g., organization, study skills, time management, digital 
literacy) is essential to ensure that students graduate from high school ready for college 
and careers. Coherent implementation across grades and subject matter as well as a 
professional culture focused on inquiry ensures academic rigor and provides a system of 
academic supports to Black and Latino males. 

• Promising youth development programs and advising strategies promote self-efficacy 
and self-esteem and assist students as they navigate college and career options. In 
addition, a focus on strategies that develop students’ skills in the area of course planning 
and career planning (such as high school course selection, goal setting, career exploration 
and planning, and postsecondary options) are part of youth development. Transition 
support services in the form of academic advising and mentoring programs inform 
students about postsecondary pathways and prepare students to persist and meet 
challenges ahead. 

• College and career access focuses on strategies that build pathways from high school to 
college and careers. Collaboration between secondary and postsecondary institutions in 
course offerings and other programs help students transition to college and careers. 
Examples of such programs might include dual enrollment, programs that offer job 
shadowing opportunities and career-related field trips, or intentionally designed 
approaches that link articulated sequences of rigorous courses to on-site work 
opportunities and other forms of contextual learning opportunities. 
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Research Design: 

Both the implementation and impact components will extend over four and a half years, 
beginning June 1, 2012 through December 31, 2016. This timeframe will enable the evaluation 
team to observe and measure the development, implementation, and refinement of the ESI over 
four school years. It will also allow us to follow and assess ESI impacts for at least one cohort of 
ninth grade students through their scheduled high school graduation in June 2016. We also plan 
to follow three additional cohorts of ninth grade students through the 2015-2016 school year. 

Implementation Study 

The implementation study design includes multiple qualitative data collections over the course of 
ESI funding and programming. Researchers will use schools’ design challenge applications to 
create school portfolios focused on the three main components specified in the design challenge. 
Researchers will also conduct interviews, focus groups, and document collection to continue to 
build portfolios that describe ESI-related programs. The qualitative research and portfolios will 
provide rich information about implementation and process. In addition, researchers will design 
rubrics aligned with the three main components to score school portfolios. Rubrics will include 
ratings about the level of evidence, quality, and frequency as a way to quantify level of 
implementation fidelity to the schools’ ESI-related programs. We will also construct portfolios 
for schools that applied to but did not receive ESI funding to provide a contrast of service 
availability and intensity for ESI and comparison schools. 

Impact Study 

We propose to employ two complementary research designs that can yield high quality 
comparison groups and valid impact estimates. First, we will use the High School Application 
and Placement (HSAP) process to identify individual students who enroll in ESI schools and 
comparison students who expressed interest in attending one of these schools but were matched 
with a non-ESI school. This approach has been used recently by MDRC in its widely cited 
evaluation of New York City’s small high schools of choice. The viability of this approach will 
depend, in part, on whether the ESI schools are oversubscribed. Second, to ensure that the impact 
study includes valid comparison groups for all ESI schools we will use and adapt comparative 
interrupted time series (CITS) analyses to compare the performance of students in the ESI 
schools with that of their counterparts who were enrolled in the same schools prior to ESI. This 
design also includes a comparison with similar students in other similar non-ESI schools during 
the same time periods. 

Data Collection and Analysis: 

Implementation Study 

Preliminary data collection activities will begin in the summer and fall of 2012 before ESI 
programs have been fully implemented. These will include (1) a systematic review of ESI 
applications and school selection criteria, (2) interviews with principals and planning team 
members from selected ESI schools to learn about planning activities, and (3) visits to selected 
ESI schools to collect data on existing services, ESI planning and early implementation 
activities. In the spring semester of each year of the study, the research team will visit all 40 ESI 
schools to collect data on ESI implementation fidelity, service availability, and intensity. Each 
spring, we will also visit 40 comparison schools to collect data on availability and quality of 
services that may be similar to those intended to be provided through ESI. Field research will 


SREE Fall 2012 Conference Abstract Template 


3 



include highly structured interviews with the principal and purposefully selected members of the 
ESI planning teams. We will examine data related to the three components that comprise the 
theory of action: academic coherence, youth development, and college and career access. 

Impact Study 

The student impact analysis will draw primarily on data from the administrative records archive 
that the Research Alliance for New York City compiles in collaboration with the DOE. Table 1 
provides a list of the key data elements and measures that we will utilize for this evaluation 
(Please insert Table 1 here.) 

• Administrative records will provide information about school enrollment status and 
academic performance throughout high school. Information from students’ middle school 
years will account for differences in the achievement and performance that students bring 
into high school and construct subgroups that may benefit differentially from ESI. The 
Research Alliance’s archive on administrative records will allow us to construct a range 
of college preparation outcomes based both on traditional measures such as credit 
accumulation and grade point averages, and on configurations of Regents examinations 
and Advanced Placement courses that may be stronger predictors of college-readiness. 

• National Student Clearinghouse records on college enrollment and persistence and 
with the City University of New York (CUNY) to obtain access to the college transcripts 
of DOE students who enroll at 2- and 4-year CUNY schools. 

• Student surveys in both ESI and non-ESI comparison schools at two points in their high 
school careers. Student surveys will cover four overarching measurement domains: (1) 
perceptions of school climate and emphasis on college and career preparation, (2) youth 
development and social-emotional competencies, (3) college knowledge, and (4) 
employment and career preparation. We will calculate mean scores for individual 
students as well as school-level scores to understand the context, climate, and initiatives 
at each of the schools. We can also examine differences in responses within schools (a 
measure of coherence) and between schools to understand more about the strategies at 
each of the 40 schools. (Please insert Table 2 here). 

Findings / Results: 

Because we will collect youth development, college knowledge, and career development 
measures for both students in ESI schools and students in the non-ESI comparison schools, we 
can determine whether the ESI is having an impact on individual student aspirations and 
pathways to college. These aspects provide a much-needed balance to the more traditional focus 
on academic preparation and achievement as well as the primary indicators of college readiness. 
In addition, and most importantly, because we will have this information for individual students 
we will be able to determine whether the ESI impacts are different for Black and Latino boys 
than they are for other groups. Preliminary results will be released following the first year of data 
collection in 2013. 

The qualitative data in the implementation component of this evaluation will help us understand 
more about which strategies are successful, how they are implemented at the school level, the 
barriers to their success, and how these barriers might be addressed. These findings will allow us 
to make policy recommendations that will inform efforts to scale up best practices around 
college and career readiness across the system. 
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Appendix B. Tables and Figures 


Table 1 

Expanded Success Initiative Evaluation: Administrative Records Data and Measures 


Data Source 

Measures 

Administrative Records: 

Student background information 

• Gender 

• Race/Ethnicity 

• Home language 

• Immigrant status 

• Residence zip code, sub district or census 
block 

Administrative Records: 

School enrollment/special service information 

• Enrollment status (active, dropout, 
discharge, etc...) 

• School transfers (grades 6-12) 

• Daily attendance (grades 6-12) 

• English language learning services (grades 
6-12) 

• Special education services (grades 6-12) 

Administrative Records: 
Achievement data 

• New York State math and English 
language arts test scores (grades 6-8) 

• Regents Examination scores 

• PSAT, SAT or ACT test scores 

Administrative Records: 

High school performance and graduation 
information 

• Credit accumulation 

• Course grades and GPA 

• Graduation status and diploma type 

National Student Clearinghouse 
(tentative) 

• Annual college enrollment status (covers 
approximately 95 percent of US 2- and 4- 
year colleges) 

• Degree attainment 

City University of New York 
(tentative) 

• Annual college enrollment status 

• Remedial course taking 

• Credit accumulation 

• Course grades and grade point averages 

• Degree attainment 

• Financial aid qualifications and receipt 
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Table 2 


Expanded Success Initiative: Student Survey Measures 


Measurement Domain 

Measures and Indicators 

School climate and emphasis on college and 
career preparation 

• Communication with adults 

• Expectations 

• Peer relationships 

Youth development, social-emotional 
competencies, and civic engagement 

• Grit 

• Persistence 

• Response to challenge 

• Cooperation and communication 

• Empathy 

• Volunteer experiences 

College knowledge 

• Familiarity with college application 
processes and financial aid 

• Knowledge of local postsecondary 
educational opportunities 

• Experience on college campuses 

Employment and career development 

• High school work experiences 

• Participation in work-related learning 
activities 

• Exposure to “soft skills” development 
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